Lecture 25 - Your project. Deep Learning with H2O. P.1 Install and explore by example
Deep Learning… In this lecture I would like to esplore with you the DeepLearning we can do with H2O package in R…
Understanding architecture
The ‘Thing’ will be working on your computer. In fact you will install another ‘computer’ inside your ‘computer’…
Installing from R and starting it up
These instructions I got from: (http://h2o.ai/download/)[http://h2o.ai/download/]. Simply use ‘Install from R’ option
# The following two commands will remove any previously installed H2O packages for R.
if ("package:h2o" %in% search()) { detach("package:h2o", unload=TRUE) }
if ("h2o" %in% rownames(installed.packages())) { remove.packages("h2o") }
# Next, we download packages that H2O depends on.
pkgs <- c("statmod","RCurl","jsonlite")
for (pkg in pkgs) {
if (! (pkg %in% rownames(installed.packages()))) { install.packages(pkg) }
}
# Now we download, install and initialize the H2O package for R.
install.packages("h2o", type="source", repos="http://h2o-release.s3.amazonaws.com/h2o/rel-weierstrass/7/R")
# Finally, let's load H2O and start up an H2O cluster
library(h2o)
h2o.init()
As you see this will actually start the ‘cluster’ on our machine! We will now can look on Localhost:54321 to see the ‘interface’ of our cluster…
As a trial let’s learn how to shut down the machine…
# we can shut down the 'machine' like this...
h2o.shutdown(prompt= FALSE)
In this case I have installed H2O Machine Learning Platform on my PC, but in the real world you may install it on more powerfull computer.
Explore the ‘thing’ on example
We will now run the demo from H2O. Our goal will be to teach system what is normal by using ECG dataset. After that we will use another dataset that will contain anomaly and use our Deep Learning model to detect that
Start Virtual Machine h2o
First thing we will launch the machine again…
# to load the library
library(h2o)
# to initialize the 'machine'
h2o.init()
H2O is not running yet, starting it now...
Note: In case of errors look at the following log files:
C:\Users\fxtrams\AppData\Local\Temp\RtmpOi23u1/h2o_fxtrams_started_from_r.out
C:\Users\fxtrams\AppData\Local\Temp\RtmpOi23u1/h2o_fxtrams_started_from_r.err
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
Starting H2O JVM and connecting: . Connection successful!
R is connected to the H2O cluster:
H2O cluster uptime: 2 seconds 152 milliseconds
H2O cluster version: 3.14.0.7
H2O cluster version age: 22 days
H2O cluster name: H2O_started_from_R_fxtrams_zrj221
H2O cluster total nodes: 1
H2O cluster total memory: 1.77 GB
H2O cluster total cores: 4
H2O cluster allowed cores: 4
H2O cluster healthy: TRUE
H2O Connection ip: localhost
H2O Connection port: 54321
H2O Connection proxy: NA
H2O Internal Security: FALSE
H2O API Extensions: Algos, AutoML, Core V3, Core V4
R Version: R version 3.2.5 (2016-04-14)
Load datasets from H2O
Then we will download the datasets from h2o.
# Import ECG train and test data into the H2O cluster
train_ecg <- h2o.importFile(
path = "http://h2o-public-test-data.s3.amazonaws.com/smalldata/anomaly/ecg_discord_train.csv",
header = FALSE,
sep = ",")
|
| | 0%
|
|================================================================================================================| 100%
test_ecg <- h2o.importFile(
path = "http://h2o-public-test-data.s3.amazonaws.com/smalldata/anomaly/ecg_discord_test.csv",
header = FALSE,
sep = ",")
|
| | 0%
|
|================================================================================================================| 100%
Understanding dataset
Personally I like to know what is in the data and how it look like!!! I will just use this link and put it into the browser. This will download the files with raw data. If I open this dataset it would not tell me much! I will plot this data in excel…
Train data set 
Test data set 
Or, I can also pull the data from h2o into R and make some 3D visualizations…
library(tidyverse)
# data frame matrix for training dataset
matrix_train <- train_ecg %>% as.data.frame() %>% as.matrix.data.frame()
# data frame matrix for test dataset
matrix_test <- test_ecg %>% as.data.frame() %>% as.matrix.data.frame()
from there we can see that the difference between both are in the three new rows 21-23
# using library plotly to plot 3D surface
library(plotly)
plot_ly(z = matrix_train, type = "surface")
# using library plotly to plot 3D surface
plot_ly(z = matrix_test, type = "surface")
Building Deep Learning model with autoencoder
Now, once we know how our data looks like we can start to do our Anomaly Model
# Train deep autoencoder learning model on "normal"
# training data, y ignored
anomaly_model <- h2o.deeplearning(
x = names(train_ecg),
training_frame = train_ecg,
activation = "Tanh",
autoencoder = TRUE,
hidden = c(50,20,50),
sparse = TRUE,
l1 = 1e-4,
epochs = 100)
|
| | 0%
|
|================================================================================================================| 100%
Calculate MSE from the train dataset
Let’s use this model on our training dataset…
# computer error of the model
mod_error <- h2o.anomaly(anomaly_model, train_ecg)
get it as a plot and see that the value is very low
# visually see it
h2o.anomaly(anomaly_model, train_ecg) %>%
as.data.frame() %>% plot.ts(ylim = c(0, 2))
Detect Anomaly using MSE value
Once our model is made, we can use it to detect anomalies in our test dataset
# Compute reconstruction error with the Anomaly
# detection app (MSE between output and input layers)
recon_error <- h2o.anomaly(anomaly_model, test_ecg)
# Pull reconstruction error data into R and
# plot to find outliers (last 3 heartbeats)
df_recon_error <- as.data.frame(recon_error)
tail(df_recon_error, 9)
We can plot this as well
plot.ts(df_recon_error, type = "p")
abline(h = 2, col = "red")
What it is telling us is that based on the new data we have anomaly in elements 21, 22, 23
Use Anomaly model to reconstruct Test Dataset
Now we can obtain predictions, or physical values using our model. We should provide the model and test dataset
# Note: Testing = Reconstructing the test dataset
test_recon <- h2o.predict(anomaly_model, test_ecg)
In order to make things visible. Once again I ask excel to help… Here I simply write teh dataframe to csv and create graph in excel
# write to csv to use it in excel
test_recon %>% as.data.frame() %>%
write.csv("test_predicted.csv")
Predicted data set 
Visualize as 3D
Or we can visualize in 3D directly in R
# making a matrix dataframe
recon_matrix <- test_recon %>% as.data.frame() %>% as.matrix.data.frame()
# make 3D plot
plot_ly(z = recon_matrix, type = "surface")
Saving Deep Learning Model for the future use
To use our model in our ShinyApp we will save it…
h2o.saveModel(anomaly_model, "C:/Users/fxtrams/Downloads/tmp/anomaly_model.bin")
h2o.download_pojo(anomaly_model, "C:/Users/fxtrams/Downloads/tmp", get_jar = TRUE)
And let’s not forget to switch off our cluster!
h2o.shutdown(prompt= FALSE)
Conclusion
In this example the Anomaly Detection model was able to output the anomaly in rows 21-23.
It learned on the pattern of many vectors and was able to distinguish the anomaly coming on new dataset
Practical use of this model can be to us function h2o.anomaly. In case the MSE value will be high - the anomaly is detected!
Next step
our next step will be to repeat the procedure but on our machine data.
- re-arranging data as matrix
- fitting deep learning models
- testing the models
- saving models
- implementation in our ShinyApp…
LS0tDQp0aXRsZTogIkxlY3R1cmUgMjUgLSBJbnRvIERlZXAgTGVhcm5pbmciDQpvdXRwdXQ6DQogIGh0bWxfZG9jdW1lbnQ6IGRlZmF1bHQNCiAgaHRtbF9ub3RlYm9vazogZGVmYXVsdA0KICBwZGZfZG9jdW1lbnQ6IGRlZmF1bHQNCi0tLQ0KDQpUaGlzIGlzIGFuIFtSIE1hcmtkb3duXShodHRwOi8vcm1hcmtkb3duLnJzdHVkaW8uY29tKSBOb3RlYm9vaw0KDQoNCiMjIyBMZWN0dXJlIDI1IC0gWW91ciBwcm9qZWN0LiBEZWVwIExlYXJuaW5nIHdpdGggSDJPLiBQLjEgSW5zdGFsbCBhbmQgZXhwbG9yZSBieSBleGFtcGxlDQoNCkRlZXAgTGVhcm5pbmcuLi4gSW4gdGhpcyBsZWN0dXJlIEkgd291bGQgbGlrZSB0byBlc3Bsb3JlIHdpdGggeW91IHRoZSBEZWVwTGVhcm5pbmcgd2UgY2FuIGRvIHdpdGggSDJPIHBhY2thZ2UgaW4gUi4uLg0KDQojIyMjIExlYXJuaW5nIHRvIGRvIERlZXAgTGVhcm5pbmcNCg0KVXNlZCBleGFtcGxlIGZyb206IChodHRwczovL2R6b25lLmNvbS9hcnRpY2xlcy9hbm9tYWx5LWRldGVjdGlvbi13aXRoLWRlZXAtbGVhcm5pbmctaW4tci13aXRoLWgybylbaHR0cHM6Ly9kem9uZS5jb20vYXJ0aWNsZXMvYW5vbWFseS1kZXRlY3Rpb24td2l0aC1kZWVwLWxlYXJuaW5nLWluLXItd2l0aC1oMm9dDQoNCk1vcmUgcmVhZGluZzogKGh0dHBzOi8vZHpvbmUuY29tL2FydGljbGVzL3RoZS1iYXNpY3Mtb2YtZGVlcC1sZWFybmluZy1ob3ctdG8tYXBwbHktaXQtdG8tcHJlP2Zyb21yZWw9dHJ1ZSlbaHR0cHM6Ly9kem9uZS5jb20vYXJ0aWNsZXMvdGhlLWJhc2ljcy1vZi1kZWVwLWxlYXJuaW5nLWhvdy10by1hcHBseS1pdC10by1wcmU/ZnJvbXJlbD10cnVlXQ0KDQpBbmQ6IChodHRwczovL3NoaXJpbmcuZ2l0aHViLmlvL21hY2hpbmVfbGVhcm5pbmcvMjAxNy8wNS8wMS9mcmF1ZClbaHR0cHM6Ly9zaGlyaW5nLmdpdGh1Yi5pby9tYWNoaW5lX2xlYXJuaW5nLzIwMTcvMDUvMDEvZnJhdWRdDQoNCkluIHRoaXMgbGVjdHVyZSB3ZSB3b3VsZCBleHBsb3JlIHRoZSAndGVjaG5vbG9neScgb24gdGhlIHNhbXBsZSBhbmQgdHJ5IHRvIGRvIHRoaXMgaW4gMTAgbWluIGxlY3R1cmUhDQoNCiMjIyMgVW5kZXJzdGFuZGluZyBhcmNoaXRlY3R1cmUNCg0KVGhlICdUaGluZycgd2lsbCBiZSB3b3JraW5nIG9uIHlvdXIgY29tcHV0ZXIuIEluIGZhY3QgeW91IHdpbGwgaW5zdGFsbCBhbm90aGVyICdjb21wdXRlcicgaW5zaWRlIHlvdXIgJ2NvbXB1dGVyJy4uLg0KDQojIyMjIEluc3RhbGxpbmcgZnJvbSBSIGFuZCBzdGFydGluZyBpdCB1cA0KDQoNClRoZXNlIGluc3RydWN0aW9ucyBJIGdvdCBmcm9tOiAoaHR0cDovL2gyby5haS9kb3dubG9hZC8pW2h0dHA6Ly9oMm8uYWkvZG93bmxvYWQvXS4gU2ltcGx5IHVzZSAnSW5zdGFsbCBmcm9tIFInIG9wdGlvbg0KDQpgYGB7ciwgZXZhbD1GQUxTRSwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRSwgaW5jbHVkZT1UUlVFfQ0KIyBUaGUgZm9sbG93aW5nIHR3byBjb21tYW5kcyB3aWxsIHJlbW92ZSBhbnkgcHJldmlvdXNseSBpbnN0YWxsZWQgSDJPIHBhY2thZ2VzIGZvciBSLg0KaWYgKCJwYWNrYWdlOmgybyIgJWluJSBzZWFyY2goKSkgeyBkZXRhY2goInBhY2thZ2U6aDJvIiwgdW5sb2FkPVRSVUUpIH0NCmlmICgiaDJvIiAlaW4lIHJvd25hbWVzKGluc3RhbGxlZC5wYWNrYWdlcygpKSkgeyByZW1vdmUucGFja2FnZXMoImgybyIpIH0NCg0KIyBOZXh0LCB3ZSBkb3dubG9hZCBwYWNrYWdlcyB0aGF0IEgyTyBkZXBlbmRzIG9uLg0KcGtncyA8LSBjKCJzdGF0bW9kIiwiUkN1cmwiLCJqc29ubGl0ZSIpDQpmb3IgKHBrZyBpbiBwa2dzKSB7DQppZiAoISAocGtnICVpbiUgcm93bmFtZXMoaW5zdGFsbGVkLnBhY2thZ2VzKCkpKSkgeyBpbnN0YWxsLnBhY2thZ2VzKHBrZykgfQ0KfQ0KDQojIE5vdyB3ZSBkb3dubG9hZCwgaW5zdGFsbCBhbmQgaW5pdGlhbGl6ZSB0aGUgSDJPIHBhY2thZ2UgZm9yIFIuDQppbnN0YWxsLnBhY2thZ2VzKCJoMm8iLCB0eXBlPSJzb3VyY2UiLCByZXBvcz0iaHR0cDovL2gyby1yZWxlYXNlLnMzLmFtYXpvbmF3cy5jb20vaDJvL3JlbC13ZWllcnN0cmFzcy83L1IiKQ0KDQojIEZpbmFsbHksIGxldCdzIGxvYWQgSDJPIGFuZCBzdGFydCB1cCBhbiBIMk8gY2x1c3Rlcg0KbGlicmFyeShoMm8pDQpoMm8uaW5pdCgpDQpgYGANCg0KQXMgeW91IHNlZSB0aGlzIHdpbGwgYWN0dWFsbHkgc3RhcnQgdGhlICdjbHVzdGVyJyBvbiBvdXIgbWFjaGluZSEgV2Ugd2lsbCBub3cgY2FuIGxvb2sgb24gYExvY2FsaG9zdDo1NDMyMWAgdG8gc2VlIHRoZSAnaW50ZXJmYWNlJyBvZiBvdXIgY2x1c3Rlci4uLg0KDQpBcyBhIHRyaWFsIGxldCdzIGxlYXJuIGhvdyB0byBzaHV0IGRvd24gdGhlIG1hY2hpbmUuLi4NCg0KYGBge3IsIGV2YWw9RkFMU0UsIGluY2x1ZGU9VFJVRX0NCiMgd2UgY2FuIHNodXQgZG93biB0aGUgJ21hY2hpbmUnIGxpa2UgdGhpcy4uLg0KaDJvLnNodXRkb3duKHByb21wdD0gRkFMU0UpDQpgYGANCg0KSW4gdGhpcyBjYXNlIEkgaGF2ZSBpbnN0YWxsZWQgSDJPIE1hY2hpbmUgTGVhcm5pbmcgUGxhdGZvcm0gb24gbXkgUEMsIGJ1dCBpbiB0aGUgcmVhbCB3b3JsZCB5b3UgbWF5IGluc3RhbGwgaXQgb24gbW9yZSBwb3dlcmZ1bGwgY29tcHV0ZXIuDQoNCiMjIyMgRXhwbG9yZSB0aGUgJ3RoaW5nJyBvbiBleGFtcGxlDQoNCldlIHdpbGwgbm93IHJ1biB0aGUgZGVtbyBmcm9tIEgyTy4gT3VyIGdvYWwgd2lsbCBiZSB0byB0ZWFjaCBzeXN0ZW0gd2hhdCBpcyBgbm9ybWFsYCBieSB1c2luZyBFQ0cgZGF0YXNldC4gQWZ0ZXIgdGhhdCB3ZSB3aWxsIHVzZSBhbm90aGVyIGRhdGFzZXQgdGhhdCB3aWxsIGNvbnRhaW4gYGFub21hbHlgIGFuZCB1c2Ugb3VyIERlZXAgTGVhcm5pbmcgbW9kZWwgdG8gZGV0ZWN0IHRoYXQNCg0KIyMjIyBTdGFydCBWaXJ0dWFsIE1hY2hpbmUgaDJvDQoNCkZpcnN0IHRoaW5nIHdlIHdpbGwgbGF1bmNoIHRoZSBtYWNoaW5lIGFnYWluLi4uDQoNCmBgYHtyLCBldmFsPVRSVUUsIG1lc3NhZ2U9RkFMU0UsIHdhcm5pbmc9RkFMU0UsIGluY2x1ZGU9VFJVRX0NCiMgdG8gbG9hZCB0aGUgbGlicmFyeQ0KbGlicmFyeShoMm8pDQoNCiMgdG8gaW5pdGlhbGl6ZSB0aGUgJ21hY2hpbmUnDQpoMm8uaW5pdCgpDQpgYGANCg0KIyMjIyBMb2FkIGRhdGFzZXRzIGZyb20gSDJPDQoNClRoZW4gd2Ugd2lsbCBkb3dubG9hZCB0aGUgZGF0YXNldHMgZnJvbSBoMm8uDQoNCmBgYHtyLCBldmFsPVRSVUUsIGluY2x1ZGU9VFJVRX0NCiMgSW1wb3J0IEVDRyB0cmFpbiBhbmQgdGVzdCBkYXRhIGludG8gdGhlIEgyTyBjbHVzdGVyDQp0cmFpbl9lY2cgPC0gaDJvLmltcG9ydEZpbGUoDQogcGF0aCA9ICJodHRwOi8vaDJvLXB1YmxpYy10ZXN0LWRhdGEuczMuYW1hem9uYXdzLmNvbS9zbWFsbGRhdGEvYW5vbWFseS9lY2dfZGlzY29yZF90cmFpbi5jc3YiLCANCiBoZWFkZXIgPSBGQUxTRSwgDQogc2VwID0gIiwiKQ0KdGVzdF9lY2cgPC0gaDJvLmltcG9ydEZpbGUoDQogcGF0aCA9ICJodHRwOi8vaDJvLXB1YmxpYy10ZXN0LWRhdGEuczMuYW1hem9uYXdzLmNvbS9zbWFsbGRhdGEvYW5vbWFseS9lY2dfZGlzY29yZF90ZXN0LmNzdiIsIA0KIGhlYWRlciA9IEZBTFNFLCANCiBzZXAgPSAiLCIpDQpgYGANCg0KIyMjIyBVbmRlcnN0YW5kaW5nIGRhdGFzZXQNCg0KUGVyc29uYWxseSBJIGxpa2UgdG8ga25vdyB3aGF0IGlzIGluIHRoZSBkYXRhIGFuZCBob3cgaXQgbG9vayBsaWtlISEhIEkgd2lsbCBqdXN0IHVzZSB0aGlzIGxpbmsgYW5kIHB1dCBpdCBpbnRvIHRoZSBicm93c2VyLiBUaGlzIHdpbGwgZG93bmxvYWQgdGhlIGZpbGVzIHdpdGggcmF3IGRhdGEuIElmIEkgb3BlbiB0aGlzIGRhdGFzZXQgaXQgd291bGQgbm90IHRlbGwgbWUgbXVjaCEgSSB3aWxsIHBsb3QgdGhpcyBkYXRhIGluIGV4Y2VsLi4uDQoNCioqVHJhaW4gZGF0YSBzZXQqKg0KIVtUcmFpbiBEYXRhXVtpZDFdDQoNCioqVGVzdCBkYXRhIHNldCoqDQohW1Rlc3QgRGF0YV1baWQyXQ0KDQpPciwgSSBjYW4gYWxzbyBwdWxsIHRoZSBkYXRhIGZyb20gaDJvIGludG8gUiBhbmQgbWFrZSBzb21lIDNEIHZpc3VhbGl6YXRpb25zLi4uDQoNCmBgYHtyLCBldmFsPVRSVUUsIGluY2x1ZGU9VFJVRX0NCmxpYnJhcnkodGlkeXZlcnNlKQ0KIyBkYXRhIGZyYW1lIG1hdHJpeCBmb3IgdHJhaW5pbmcgZGF0YXNldA0KbWF0cml4X3RyYWluIDwtIHRyYWluX2VjZyAlPiUgYXMuZGF0YS5mcmFtZSgpICU+JSBhcy5tYXRyaXguZGF0YS5mcmFtZSgpDQojIGRhdGEgZnJhbWUgbWF0cml4IGZvciB0ZXN0IGRhdGFzZXQNCm1hdHJpeF90ZXN0IDwtIHRlc3RfZWNnICU+JSBhcy5kYXRhLmZyYW1lKCkgJT4lIGFzLm1hdHJpeC5kYXRhLmZyYW1lKCkNCmBgYA0KDQpmcm9tIHRoZXJlIHdlIGNhbiBzZWUgdGhhdCB0aGUgZGlmZmVyZW5jZSBiZXR3ZWVuIGJvdGggYXJlIGluIHRoZSB0aHJlZSBuZXcgcm93cyAyMS0yMw0KDQpgYGB7ciwgZXZhbD1GQUxTRSwgaW5jbHVkZT1UUlVFfQ0KIyB1c2luZyBsaWJyYXJ5IHBsb3RseSB0byBwbG90IDNEIHN1cmZhY2UNCmxpYnJhcnkocGxvdGx5KQ0KcGxvdF9seSh6ID0gbWF0cml4X3RyYWluLCB0eXBlID0gInN1cmZhY2UiKQ0KDQpgYGANCg0KIVtUcmFpbiBEYXRhc2V0IHdpdGggcGxvdGx5XVtpZDRdDQoNCmBgYHtyLCBldmFsPUZBTFNFLCBpbmNsdWRlPVRSVUV9DQojIHVzaW5nIGxpYnJhcnkgcGxvdGx5IHRvIHBsb3QgM0Qgc3VyZmFjZQ0KcGxvdF9seSh6ID0gbWF0cml4X3Rlc3QsIHR5cGUgPSAic3VyZmFjZSIpDQoNCmBgYA0KDQohW1Rlc3QgRGF0YXNldCB3aXRoIHBsb3RseV1baWQ1XQ0KDQojIyMjIEJ1aWxkaW5nIERlZXAgTGVhcm5pbmcgbW9kZWwgd2l0aCBhdXRvZW5jb2Rlcg0KDQpOb3csIG9uY2Ugd2Uga25vdyBob3cgb3VyIGRhdGEgbG9va3MgbGlrZSB3ZSBjYW4gc3RhcnQgdG8gZG8gb3VyIEFub21hbHkgTW9kZWwNCg0KYGBge3IsIGV2YWw9VFJVRSwgaW5jbHVkZT1UUlVFfQ0KDQojIFRyYWluIGRlZXAgYXV0b2VuY29kZXIgbGVhcm5pbmcgbW9kZWwgb24gIm5vcm1hbCIgDQojIHRyYWluaW5nIGRhdGEsIHkgaWdub3JlZCANCmFub21hbHlfbW9kZWwgPC0gaDJvLmRlZXBsZWFybmluZygNCiB4ID0gbmFtZXModHJhaW5fZWNnKSwgDQogdHJhaW5pbmdfZnJhbWUgPSB0cmFpbl9lY2csIA0KIGFjdGl2YXRpb24gPSAiVGFuaCIsIA0KIGF1dG9lbmNvZGVyID0gVFJVRSwgDQogaGlkZGVuID0gYyg1MCwyMCw1MCksIA0KIHNwYXJzZSA9IFRSVUUsDQogbDEgPSAxZS00LCANCiBlcG9jaHMgPSAxMDApDQpgYGANCg0KIyMjIyBDYWxjdWxhdGUgTVNFIGZyb20gdGhlIHRyYWluIGRhdGFzZXQNCg0KTGV0J3MgdXNlIHRoaXMgbW9kZWwgb24gb3VyIHRyYWluaW5nIGRhdGFzZXQuLi4NCg0KYGBge3IsIGV2YWw9VFJVRSwgaW5jbHVkZT1UUlVFfQ0KIyBjb21wdXRlciBlcnJvciBvZiB0aGUgbW9kZWwNCm1vZF9lcnJvciA8LSBoMm8uYW5vbWFseShhbm9tYWx5X21vZGVsLCB0cmFpbl9lY2cpDQpgYGANCg0KZ2V0IGl0IGFzIGEgcGxvdCBhbmQgc2VlIHRoYXQgdGhlIHZhbHVlIGlzIHZlcnkgbG93DQoNCmBgYHtyLCBldmFsPVRSVUUsIGluY2x1ZGU9VFJVRX0NCiMgdmlzdWFsbHkgc2VlIGl0DQpoMm8uYW5vbWFseShhbm9tYWx5X21vZGVsLCB0cmFpbl9lY2cpICU+JSANCiAgYXMuZGF0YS5mcmFtZSgpICU+JSBwbG90LnRzKHlsaW0gPSBjKDAsIDIpKQ0KYGBgDQoNCiMjIyMgRGV0ZWN0IEFub21hbHkgdXNpbmcgTVNFIHZhbHVlDQoNCk9uY2Ugb3VyIG1vZGVsIGlzIG1hZGUsIHdlIGNhbiB1c2UgaXQgdG8gZGV0ZWN0IGFub21hbGllcyBpbiBvdXIgKip0ZXN0KiogZGF0YXNldA0KDQoNCmBgYHtyLCBldmFsPVRSVUUsIGluY2x1ZGU9VFJVRX0NCiMgQ29tcHV0ZSByZWNvbnN0cnVjdGlvbiBlcnJvciB3aXRoIHRoZSBBbm9tYWx5IA0KIyBkZXRlY3Rpb24gYXBwIChNU0UgYmV0d2VlbiBvdXRwdXQgYW5kIGlucHV0IGxheWVycykNCnJlY29uX2Vycm9yIDwtIGgyby5hbm9tYWx5KGFub21hbHlfbW9kZWwsIHRlc3RfZWNnKQ0KDQojIFB1bGwgcmVjb25zdHJ1Y3Rpb24gZXJyb3IgZGF0YSBpbnRvIFIgYW5kIA0KIyBwbG90IHRvIGZpbmQgb3V0bGllcnMgKGxhc3QgMyBoZWFydGJlYXRzKQ0KZGZfcmVjb25fZXJyb3IgPC0gYXMuZGF0YS5mcmFtZShyZWNvbl9lcnJvcikNCnRhaWwoZGZfcmVjb25fZXJyb3IsIDkpDQpgYGANCg0KV2UgY2FuIHBsb3QgdGhpcyBhcyB3ZWxsDQpgYGB7ciwgZXZhbD1UUlVFLCBpbmNsdWRlPVRSVUV9DQpwbG90LnRzKGRmX3JlY29uX2Vycm9yLCB0eXBlID0gInAiLCBwY2g9MjApDQphYmxpbmUoaCA9IDIsIGNvbCA9ICJyZWQiKQ0KYGBgDQoNCldoYXQgaXQgaXMgdGVsbGluZyB1cyBpcyB0aGF0IGJhc2VkIG9uIHRoZSBuZXcgZGF0YSB3ZSBoYXZlIGFub21hbHkgaW4gZWxlbWVudHMgMjEsIDIyLCAyMw0KDQojIyMjIFVzZSBBbm9tYWx5IG1vZGVsIHRvIHJlY29uc3RydWN0IFRlc3QgRGF0YXNldA0KDQpOb3cgd2UgY2FuIG9idGFpbiBwcmVkaWN0aW9ucywgb3IgcGh5c2ljYWwgdmFsdWVzIHVzaW5nIG91ciBtb2RlbC4gV2Ugc2hvdWxkIHByb3ZpZGUgdGhlIG1vZGVsIGFuZCB0ZXN0IGRhdGFzZXQNCg0KYGBge3IsIGV2YWw9VFJVRSwgaW5jbHVkZT1UUlVFfQ0KIyBOb3RlOiBUZXN0aW5nID0gUmVjb25zdHJ1Y3RpbmcgdGhlIHRlc3QgZGF0YXNldA0KdGVzdF9yZWNvbiA8LSBoMm8ucHJlZGljdChhbm9tYWx5X21vZGVsLCB0ZXN0X2VjZykgDQpgYGANCg0KSW4gb3JkZXIgdG8gbWFrZSB0aGluZ3MgdmlzaWJsZS4gT25jZSBhZ2FpbiBJIGFzayBleGNlbCB0byBoZWxwLi4uIEhlcmUgSSBzaW1wbHkgd3JpdGUgdGVoIGRhdGFmcmFtZSB0byBjc3YgYW5kIGNyZWF0ZSBncmFwaCBpbiBleGNlbA0KDQpgYGB7ciwgZXZhbD1GQUxTRSwgaW5jbHVkZT1UUlVFfQ0KIyB3cml0ZSB0byBjc3YgdG8gdXNlIGl0IGluIGV4Y2VsDQp0ZXN0X3JlY29uICU+JSBhcy5kYXRhLmZyYW1lKCkgJT4lIA0Kd3JpdGUuY3N2KCJ0ZXN0X3ByZWRpY3RlZC5jc3YiKQ0KYGBgDQoNCioqUHJlZGljdGVkIGRhdGEgc2V0KioNCiFbUHJlZGljdGVkIERhdGFdW2lkM10NCg0KIyMjIyBWaXN1YWxpemUgYXMgM0QNCg0KT3Igd2UgY2FuIHZpc3VhbGl6ZSBpbiAzRCBkaXJlY3RseSBpbiBSDQpgYGB7ciwgZXZhbD1GLCBpbmNsdWRlPVRSVUV9DQojIG1ha2luZyBhIG1hdHJpeCBkYXRhZnJhbWUNCnJlY29uX21hdHJpeCA8LSB0ZXN0X3JlY29uICU+JSBhcy5kYXRhLmZyYW1lKCkgJT4lIGFzLm1hdHJpeC5kYXRhLmZyYW1lKCkNCg0KIyBtYWtlIDNEIHBsb3QNCnBsb3RfbHkoeiA9IHJlY29uX21hdHJpeCwgdHlwZSA9ICJzdXJmYWNlIikNCmBgYA0KDQohW1ByZWRpY3RlZCBEYXRhc2V0IHdpdGggcGxvdGx5XVtpZDZdDQoNCiMjIyMgU2F2aW5nIERlZXAgTGVhcm5pbmcgTW9kZWwgZm9yIHRoZSBmdXR1cmUgdXNlDQoNClRvIHVzZSBvdXIgbW9kZWwgaW4gb3VyIFNoaW55QXBwIHdlIHdpbGwgc2F2ZSBpdC4uLg0KDQpgYGB7ciwgZXZhbD1GQUxTRSwgaW5jbHVkZT1UUlVFfQ0KaDJvLnNhdmVNb2RlbChhbm9tYWx5X21vZGVsLCAiQzovVXNlcnMvZnh0cmFtcy9Eb3dubG9hZHMvdG1wL2Fub21hbHlfbW9kZWwuYmluIikNCmgyby5kb3dubG9hZF9wb2pvKGFub21hbHlfbW9kZWwsICJDOi9Vc2Vycy9meHRyYW1zL0Rvd25sb2Fkcy90bXAiLCBnZXRfamFyID0gVFJVRSkNCmBgYA0KDQpBbmQgbGV0J3Mgbm90IGZvcmdldCB0byBzd2l0Y2ggb2ZmIG91ciBjbHVzdGVyIQ0KYGBge3IsIGV2YWw9VFJVRSwgaW5jbHVkZT1UUlVFfQ0KaDJvLnNodXRkb3duKHByb21wdD0gRkFMU0UpDQoNCmBgYA0KDQojIyMjIENvbmNsdXNpb24NCg0KSW4gdGhpcyBleGFtcGxlIHRoZSBBbm9tYWx5IERldGVjdGlvbiBtb2RlbCB3YXMgYWJsZSB0byBvdXRwdXQgdGhlIGFub21hbHkgaW4gcm93cyAyMS0yMy4gDQoNCkl0IGxlYXJuZWQgb24gdGhlIHBhdHRlcm4gb2YgbWFueSB2ZWN0b3JzIGFuZCB3YXMgYWJsZSB0byBkaXN0aW5ndWlzaCB0aGUgYW5vbWFseSBjb21pbmcgb24gbmV3IGRhdGFzZXQNCg0KUHJhY3RpY2FsIHVzZSBvZiB0aGlzIG1vZGVsIGNhbiBiZSB0byB1cyBmdW5jdGlvbiAqKmgyby5hbm9tYWx5KiouIEluIGNhc2UgdGhlIE1TRSB2YWx1ZSB3aWxsIGJlIGhpZ2ggLSB0aGUgYW5vbWFseSBpcyBkZXRlY3RlZCENCg0KIyMjIyBOZXh0IHN0ZXANCg0Kb3VyIG5leHQgc3RlcCB3aWxsIGJlIHRvIHJlcGVhdCB0aGUgcHJvY2VkdXJlIGJ1dCBvbiBvdXIgbWFjaGluZSBkYXRhLiANCg0KKiByZS1hcnJhbmdpbmcgZGF0YSBhcyBtYXRyaXgNCiogZml0dGluZyBkZWVwIGxlYXJuaW5nIG1vZGVscw0KKiB0ZXN0aW5nIHRoZSBtb2RlbHMNCiogc2F2aW5nIG1vZGVscw0KKiBpbXBsZW1lbnRhdGlvbiBpbiBvdXIgU2hpbnlBcHAuLi4NCg0KW2lkMV06IHRyYWluLlBORyAiVHJhaW4iDQpbaWQyXTogdGVzdC5QTkcgIlRlc3QiDQpbaWQzXTogcHJlZGljdGVkLlBORyAiUHJlZGljdGVkIg0KW2lkNF06IGgyb19kYXRhc2V0cy90cmFpbi5wbmcgIlRyYWluIg0KW2lkNV06IGgyb19kYXRhc2V0cy90ZXN0LnBuZyAiVGVzdCINCltpZDZdOiBoMm9fZGF0YXNldHMvcHJlZGljdC5wbmcgIlByZWRpY3RlZCINCg==